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DETAILED ACTION 

This non-final office action is responsive to the Request for Continued 
Examination (RCE) filed September 3, 2010. Claims 10, 20, 22, 27 and 28 are currently 
amended, and claim 21 is additionally canceled. Therefore claims 10, 11, 15, 17, 18, 
20, 22, 24-28 and 30-37 remain pending in this application. 

Claim 36 depends on canceled claim 21. For the purposes of compact 
prosecution, the Examiner will assume claim 36 depends on claim 10 and 
examine it as such. Appropriate correction is required. 

Claim Rejections - 35 USC §112 

1 . The following is a quotation of the first paragraph of 35 U.S.C. 112: 

The specification shall contain a written description of the invention, and of the manner and process of 
making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the 
art to which it pertains, or with which it is most nearly connected, to make and use the same and shall 
set forth the best mode contemplated by the inventor of carrying out his invention. 

2. Claims 22, 24, 27, 28 and 31 are rejected under 35 U.S.C. 112, first paragraph, 
as failing to comply with the written description requirement. The claim(s) contains 
subject matter which was not described in the specification in such a way as to 
reasonably convey to one skilled in the relevant art that the inventor(s), at the time the 
application was filed, had possession of the claimed invention. Claims 22, 27 and 28 
have been amended to include a limitation of a first pointer between the context 
information and content (or textual region) and a second pointer between the additional 
context information and content (or textual region). Applicant points to paragraph 59 of 
the original specification to support this amendment. 
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3. However, paragraph 59 does not provide sufficient support for the new limitation. 
Applicant's specification supports a first pointer 903 that points from context 902 to 
content 901 and a second pointer 904 that points from content 901 to context 902. 
Although Applicant describes two different pointers, the pointers described by Applicant 
are not the same as the claimed two pointers. There is no description of a second 
pointer that points between additional context information and content, as claimed. The 
pointers described in the specification only point between content and a single context 
902, not additional context. In fact, the pointers described in the specification represent 
reverse or opposite pointers between the same two objects. Therefore, the limitation of 
a first pointer between the context information and content (or textual region) and a 
second pointer between the additional context information and content (or textual 
region) represents new matter. 

4. Claims 24 and 31 are rejected as being dependent upon rejected claims 27 and 
28, respectively. 

5. The following is a quotation of the second paragraph of 35 U.S.C. 1 12: 

The specification shall conclude with one or more claims particularly pointing out and distinctly 
claiming the subject matter which the applicant regards as his invention. 

6. Claim 36 is rejected under 35 U.S.C. 112, second paragraph, as being indefinite 
for failing to particularly point out and distinctly claim the subject matter which applicant 
regards as the invention. The limitation "any subsequent sets of textual characters 
reside outside the path drawn by the user" is unclear. Any textual characters not in the 
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selected region when the region is selected are inherently outside the path drawn. 
Appropriate correction is required. 



Claim Rejections - 35 USC § 103 

7. The following is a quotation of 35 U.S.C. 1 03(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

8. Claims 10, 20, 30, 36 and 37 are rejected under 35 U.S.C. 103(a) (current 
application filing date 1/26/2004) as being unpatentable over Denoue et al. (US 
2004/01 19762 A1 , filing date 12/24/2002) ('Denoue') in view of Dawe et al. (US 
7,042,594 B1 , filing date 3/7/2000) ('Dawe'), further in view of Woolf et al. (WO 
01/42980 A1) ('Woolf), and further in view of Saund etal. (US 2003/0182630 A1) 
('Saund'). 



9. With respect to claim 10, Denoue teaches: 

setting a mode of operation to a content capture mode for interpreting user input 
for the purpose of selecting an on-screen region of a display (paragraphs 56, 75-78 and 
86), and receiving a path drawn by a user (paragraphs 54 and 86), the path defining 
boundaries of the selected region of the display (paragraph 59), wherein pixels 
comprising one or more graphical elements representing a first set of one or more 
textual characters are displayed in the selected on-screen region (Figure 6, paragraphs 
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67 and 71-72) (Denoue teaches that the word "awareness" may be a standalone object 
that may be selected by itself, and further that "increased awareness" may be looked at 
or segmented as two parts.); 

capturing the pixels displayed within the selected on-screen region (paragraphs 
67 and 86-87), and storing the captured pixels (paragraph 89); 

receiving an annotation (i.e. freeform ink) drawn by a user on the display (Figs. 2- 
5 and paragraphs 55 and 57); 

obtaining context information for the one or more graphical elements by 
automatically applying text recognition (paragraph 79) to the annotation (i.e. freeform 
inks 532, 534, 536 and freeform notes) (paragraphs 57 and 75) (OCR may be applied to 
captured content and captured content may include freeform inks, as seen in figs. 2-5); 

obtaining additional context information by extracting the first set of one or more 
textual characters and extracting a second set of textual characters displayed in 
proximity with the first set (paragraphs 71-72 and 79) (Denoue teaches a first set of 
characters "increased", which may be separate from the second set of characters 
"awareness," which is in proximity to the first set. He further teaches performing OCR 
on text to extract information.), 

wherein context information is stored in association with the captured data 
(paragraphs 75-76). 

Denoue does not explicitly teach storing captured data in an image file; storing 
the results of text recognition as context information via the computer system; or 
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automatically storing context information and additional context information in 
association with the image file. 

Dawe teaches a system and method for saving handwriting as an annotation in a 
scanned document (see abstract), in which he teaches: 

capturing image pixels and storing captured image pixels in an image file (col. 3 
lines 60-63, col. 7 line 61- col. 8 line 4); 

applying text recognition to annotations and storing the results of text recognition 
as context information via the computer system (col. 5 lines 52-60, col. 7 lines 6-20 and 
61-65); and 

automatically storing context information and additional context information in 
association with the image file (col. 7 lines 57-65). 

It would have been obvious to a person having ordinary skill in the art at the time 
the invention was made to have modified Denoue by the teaching of Dawe because 
storing captured data in an image file; storing the results of text recognition as context 
information; or automatically storing context information in association with the image 
file would enable handwriting to be saved as an annotation, and thus reduce clutter in a 
document presented to a user, while maintaining the information conveyed by the 
handwriting for presentation to the user if desired (Dawe, abstract). 

Further regarding claim 10, Denoue in view of Dawe does not teach switching the 
mode of operation to an annotation mode in response to a user command; or wherein 
the received annotation is implemented using a plurality of tools via a toolbar, the 
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toolbar appearing after selecting a button. 

Woolf teaches a system and method for annotating an electronic document 
independently of is content (see abstract), in which he teaches switching the mode of 
operation to the annotation mode in response to a user command; and 

receiving an annotation drawn by the user on the display (Figures 4 and 5, page 
7 lines 24-26), wherein the received annotation is implemented using a plurality of 
tools via a toolbar, the toolbar appearing after selecting a button (Figure 4, page 7 lines 
5-14). 

Woolf teaches a user interface on a screen that has a toolbar at the bottom of the 
screen (Figure 4). Further, the annotation menu 402 may be considered a toolbar 
because it includes interactive buttons that may be clicked to activate a tool (i.e. ink 
mode, selection mode, highlight mode or erase mode). The annotation menu appears 
in response to tapping or selecting a button. 

It would have been obvious to a person having ordinary skill in the art at the time 
the invention was made to have further modified Denoue by the teaching of Woolf to 
enable users the ability to annotate pages of a document without changing the 
contents of the document (Woolf, page 2 lines 14-15). 

Further regarding claim 10, Denoue in view of Dawe and Woolf fails to explicitly 
teach the toolbar (or menu) appearing after selecting an on screen region. 

Saund teaches a method for gestural interpretation in a system for selecting and 
arranging visible material in document images (see abstract), in which he teaches a 
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toolbar appearing after selecting an on-screen region (paragraphs 29-30). 

Saund teaches that a user draws a path that encloses or partially encloses at 
least one object. In response to this gesture, a pop-up menu choose box 320 appears. 
The pop-up menu may be considered a toolbar because the select button is an 
interactive button that may be clicked to activate a selection tool. Further, Saund 
teaches that tools are usually provided by menus (paragraph 5). 

It would have been obvious to a person having ordinary skill in the art at the time 
the invention was made to have further modified Denoue by the teaching of Saund to 
enable users the ability to enter, select or modify data without pre-selecting an input 
mode (Saund, abstract). 

With respect to claim 20, Denoue in view Dawe, Woolf and Saund teaches 
wherein the context information is stored in such a manner as to be accessible to the 
user for performing at least one of the following: 

searching for said context information; 

displaying the context information simultaneously with the captured pixels 
(Denoue, paragraphs 75-76; Dawe, col. 8 lines 1-16), and 

navigating a network to a source of the captured pixels. 

With respect to claim 30, Denoue in view Dawe, Woolf and Saund teaches 
wherein the annotation is stored as originally drawn as additional context information in 
association with the image file (Dawe, col. 7 lines 9-11,1 8-20 and 57-65). 
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With respect to claim 36, Denoue in view Dawe, Woolf and Saund teaches 
determining that the second sets of textual characters reside within the path drawn by 
the user and any subsequent sets of textual characters reside outside the path drawn 
by the user (Dawe, col. 7 lines 11-16). Dawe teaches that it is determined that the word 
"job" resides within the path drawn. It is inherent that any subsequent characters (i.e. 
characters that are not in the region when the region is selected) will be outside the path 
drawn. 

With respect to claim 37, Denoue in view Dawe, Woolf and Saund teaches 
wherein the received annotation is implemented using a plurality of tools via a toolbar 
(Woolf, Figure 4, page 7 lines 5-14), the toolbar appearing after a stylus has been lifted 
off the display (Saund, paragraphs 29-30). Saund teaches that the pop-up menu (i.e. 
toolbar) appear after a path has been drawn, meaning the stylus has been lifted. 

1 0. Claim 1 1 , 25 and 26 are rejected under 35 U.S.C. 1 03(a) as being unpatentable 
over Denoue et al. (US 2004/01 19762 A1 , filing date 12/24/2002) ('Denoue') in view of 
Dawe et al. (US 7,042,594 B1 , filing date 3/7/2000) ('Dawe'), Woolf et al. (WO 
01/42980 A1) ('Woolf'), and Saund et al. (US 2003/0182630 A1) ('Saund'), as applied to 
claims 10, 20, 30, 36 and 37 above, and further in view of Oppermann et al. (US 
6,334,157 B1, filing date 3/11/1997) ('Oppermann'). 
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With respect to claim 1 1 , Denoue in view of Dawe, Woolf and Saund teaches 
claim 10, including a selected on-screen region and automatically storing context 
information in association with the image file (Dawe, col. 7 lines 57-65). 

Denoue in view of Dawe, Woolf and Saund does not teach determining a window 
associated with the selected on-screen region; retrieving an application interface having 
a uniform resource identifier (URI) property from the determined window or parent 
window of the determined window; or obtaining the URI property as additional context 
information. 

Oppermann teaches programmatically providing direct access to user interface 
elements of an application program (see abstract), in which he teaches: 

selecting user interface elements, such as text (col. 8 lines 43 and 49-51) and 
determining a window associated with the selected elements (col. 26 lines 37-40); 

retrieving an application interface having a uniform resource identifier (URI) 
property from the determined window or parent window of the determined window (col. 
25 lines 59-62, col. 28 lines 33-39); and 

obtaining the URI property as additional context information (col. 1 1 lines 1-9, 55- 
60, col. 12 lines 55-60, col. 13 lines 51-60). 

It would have been obvious to a person having ordinary skill in the art at the time 
the invention was made to have further modified Denoue by the teaching of Oppermann 
because determining a window, retrieving an application interface having a uniform 
resource identifier (URI) property from the determined window or parent window of the 
determined window would enable accessibility aids the ability to access and manipulate 
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user interface elements of any application program without having prior knowledge of 
the application program or its interface (Oppermann, column 4 lines 27-30). 

With respect to claim 25, Denoue in view of Dawe, Woolf, Saund and 
Oppermann teaches wherein the selected on-screen region includes at least a portion 
of a displayed web page or document (Dawe, Figure 4, col. 4 lines 38-42), and the 
method further comprises: using an application programming interface (API) to query an 
application for additional context information (Oppermann, column 7 lines 36-38), the 
additional context information being automatically stored in association with the image 
file (Dawe, col. 7 lines 57-65), the queried application causing the one or more graphical 
elements to be displayed (Oppermann, column 7 lines 36-45; Denoue, paragraphs 86- 
87). 

With respect to claim 26, Denoue in view of Dawe, Woolf, Saund and 
Oppermann teaches further comprising obtaining a uniform resource identifier (URI) of 
the web page or document as the context information (Oppermann, col. 1 1 lines 1 -9, 
55-60, col. 12 lines 55-60, col. 13 lines 51-60), the URI being obtained as a result of the 
query using the API (Oppermann, column 7 lines 36-38). 

1 1 . Claims 1 5, 1 8, 32 and 33 are rejected under 35 U.S.C. 1 03(a) as being 
unpatentable over Denoue et al. (US 2004/0119762 A1, filing date 12/24/2002) 
('Denoue') in view of Dawe et al. (US 7,042,594 B1 , filing date 3/7/2000) ('Dawe'), Woolf 
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et al. (WO 01/42980 A1) ('Woolf), and Saund et al. (US 2003/0182630 A1) ('Saund'), as 
applied to claims 10, 20, 30, 36 and 37 above, and further in view of Browne et al. (US 
2004/0135815 A1 , filing date 12/15/2003) ('Browne'). 

With respect to claim 15, Denoue in view of Dawe, Woolf and Saund teaches 
claim 10. 

Denoue in view of Dawe, Woolf and Saund does not teach creating and storing a 
linking structure as the association between the image file and the context information. 

Browne teaches a method and apparatus for image metadata entry (see 
abstract), in which he teaches creating and storing a linking structure as the association 
between the image file and the context information (Figure 12, paragraph 136). 

It would have been obvious to a person having ordinary skill in the art at the time 
the invention was made to have further modified Denoue by the teaching of Browne 
because teach creating and storing a linking structure as the association between the 
image file and the context information would enable an easy and efficient method of 
classifying and storing digital images (Browne, paragraph 14). 

With respect to claim 18, Denoue in view of Dawe, Woolf, Saund and Browne 
teaches wherein the linking structure includes at least one pointer pointing to the stored 
image file or the stored content information (Browne, paragraph 136). 

With respect to claim 32, Denoue in view of Dawe, Woolf, Saund and Browne 
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teaches a link to the one or more graphical elements of the selected on-screen region, 
the link comprising one of an implicit link hidden behind the one or more graphical 
elements and an explicit link shown in conjunction with the one or more graphical 
elements (Browne, paragraph 136). {Browne teaches metadata-image associations, in 
which pointers are stored between images and metadata). 

With respect to claim 33, Denoue in view of Dawe, Woolf, Saund and Browne 
teaches a separate linking table or database entry utilizing one or more pointers to the 
one or more graphical elements (Browne, paragraph 136). (Browne teaches metadata- 
image associations, in which pointers are stored between images and metadata). 

12. Claims 22 is rejected under 35 U.S.C. 103(a) as being unpatentable over 
Denoue et al. (US 2004/01 19762 A1 , filing date 12/24/2002) ('Denoue') in view of Dawe 
et al. (US 7,042,594 B1 , filing date 3/7/2000) ('Dawe'), Woolf et al. (WO 01/42980 A1 ) 
('Woolf), and Saund et al. (US 2003/0182630 A1 ) ('Saund'), as applied to claims 10, 20, 
30, 36 and 37 above, and further in view of Barsness et al. (US 2004/0201633 A1) 
('Barsness'). 

With respect to claim 22, Denoue in view Dawe, Woolf and Saund teaches 
wherein the selected on-screen region is part of a displayed textual region, and the 
graphical elements comprise a first set of one or more textual characters displayed in 
the textual region, the method further comprising: obtaining additional context 
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information based on a second set of one or more textual characters displayed in the 
textual region (Dawe, Fig. 4, col. 7 lines 6-20 and lines 61-65; Denoue, paragraph 72), 
the additional context information being automatically stored in association with the 
image file (Dawe, col. 7 lines 57-65) (The words "job" in Dawe or "awareness" in 
Denoue may be considered the second set of textual characters). 

Denoue in view of Dawe, Woolf and Saund fails to teach obtaining a first pointer 
between the context information and the textual region, and obtaining a second pointer 
between the additional context information and the textual region, wherein the first 
pointer differs from the second pointer. 

Barsness teaches a handheld electronic book reader with annotation and usage 
tracking capabilities (see abstract), in which he teaches obtaining a first pointer between 
the context information and the textual region, and obtaining a second pointer between 
the additional context information and the textual region, wherein the first pointer differs 
from the second pointer (paragraphs 36, 49 and 61-62). {Barsness teaches a term is 
associated with a context list that has one or more context records 69, each context 
record including either a copy of a section of text or a pointer to the actual content in an 
electronic document. The term represents a textual region, and each term may be 
associated, by pointers, with different contexts of the document.) 

It would have been obvious to a person having ordinary skill in the art at the time 
the invention was made to have further modified Denoue by the teaching of Barsness to 
enable improved functionality and utility of electronic documents by enhancing the 
collaborative and annotative capabilities of electronic book readers or other similar 
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handheld devices capable of rendering electronic documents (Barsness, abstract, 
paragraph 22). 

13. Claims 32 and 35 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Denoue et al. (US 2004/01 19762 A1 , filing date 12/24/2002) ('Denoue') in view of 
Dawe et al. (US 7,042,594 B1 , filing date 3/7/2000) ('Dawe'), Woolf et al. (WO 01/42980 
A1 ) ('Woolf ), and Saund etal. (US 2003/0182630 A1) ('Saund'), as applied to claims 
10, 20, 30, 36 and 37 above, and further in view of Hertzfeld et al. (US 2002/0076109 
A1 , filing date 1/25/1999) ('Hertzfeld'). 

With respect to claim 32, Denoue in view of Dawe, Woolf and Saund teaches 
claim 10. 

Denoue in view of Dawe, Woolf and Saund does not teach a link to the one or 
more graphical elements of the selected on-screen region, the link comprising one of an 
implicit link hidden behind the one or more graphical elements and an explicit link 
shown in conjunction with the one or more graphical elements. 

Hertzfeld teaches a method and apparatus for context sensitive text recognition 
(see abstract), in which he teaches a link to the one or more graphical elements of the 
selected on-screen region, the link comprising one of an implicit link hidden behind the 
one or more graphical elements and an explicit link shown in conjunction with the one or 
more graphical elements (elements 608 and 616 in Fig. 6, paragraph 38). 

It would have been obvious to a person having ordinary skill in the art at the time 
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the invention was made to have further modified Denoue by the teaching of Hertzfeld 
to enable recognition of predefined types of text and predefined actions to be 
performed based on the types of text (Hertzfeld, abstract). 

With respect to claim 35, Denoue in view of Dawe, Woolf, Saund and Hertzfeld 
teaches wherein the one or more graphical elements comprise underlying data, 
underlying objects, and underlying associations with remote content (Hertzfeld, Fig. 6, 
paragraph 38). 

14. Claim 34 is rejected under 35 U.S.C. 103(a) as being unpatentable over Denoue 
et al. (US 2004/01 19762 A1 , filing date 12/24/2002) ('Denoue') in view of Dawe et al. 
(US 7,042,594 B1 , filing date 3/7/2000) ('Dawe'), Woolf et al. (WO 01/42980 A1 ) 
('Woolf), and Saund et al. (US 2003/0182630 A1) ('Saund'), as applied to claims 10, 20, 
30, 36 and 37 above, and further in view of Chan (US 2006/01 00849 A1 ). 

With respect to claim 34, Denoue in view of Dawe, Woolf and Saund teaches 
claim 10. 

Denoue in view of Dawe, Woolf and Saund does not teach displaying the 
obtained context information to the user as one of displaying a property in a window and 
hovering over the one or more graphical elements. 

Chan teaches a system for providing a user bilingual annotations (see abstract), 
in which he teaches displaying the obtained context information to the user as one of 
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displaying a property in a window and hovering over the one or more graphical 
elements (paragraphs 1 1 , 49 and 66) (Chan teaches that translated versions of text 
objects may be displayed when the pointer is adjacent to or overlaid by a user's 
pointer). 

It would have been obvious to a person having ordinary skill in the art at the time 
the invention was made to have further modified Denoue by the teaching of Chan to 
enable an annotation tool providing annotations in multiple languages, thereby making 
documents accessible to more users (Chan, abstract, paragraph 14). 

15. Claim 17 is rejected under 35 U.S.C. 103(a) as being unpatentable over Denoue 
et al. (US 2004/01 19762 A1 , filing date 12/24/2002) ('Denoue') in view of Dawe et al 
(US 7,042,594 B1 , filing date 3/7/2000) ('Dawe'), Woolf et al. (WO 01/42980 A1 ) 
('Woolf), Saund et al. (US 2003/0182630 A1) ('Saund'), and Browne et al. (US 
2004/0135815 A1 , filing date 12/15/2003) ('Browne'), as applied to claims 15 and 18 
above, and further in view of Newman (US 2003/0101 156 A1, filing date 11/26/2001). 

With respect to claim 17, Denoue in view of Dawe, Woolf, Saund and Browne 
teaches claim 15. 

Denoue in view of Dawe, Woolf, Saund and Browne does not teach wherein the 
linking structure is incorporated in a file separate from the stored image file and the 
stored content information. 

Newman teaches database systems and methods (see abstract), in which he 
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teaches wherein the linking structure is incorporated in a file separate from the stored 
image file and the stored content information (paragraph 16). 

It would have been obvious to a person having ordinary skill in the art at the time 
the invention was made to have further modified Denoue by the teaching of Newman 
because wherein the linking structure is incorporated in a file separate from the stored 
image file and the stored content information would enable additional information about 
image files, such as the origination device, person who created the file, and data/time 
the file was created, to be transmitted and stored along with the image files (Newman, 
paragraph 16). 

16. Claims 24 and 27 are rejected under 35 U.S.C. 1 03(a) as being unpatentable 
over Denoue et al. (US 2004/01 19762 A1, filing date 12/24/2002) ('Denoue') in view of 
Dawe et al (US 7,042,594 B1 , filing date 3/7/2000) ('Dawe'), further in view of Hertzfeld 
et al. (US 2002/0076109 A1 , filing date 1/25/1999) ('Hertzfeld'), further in view of Chan 
(US 2006/01 00849 A1 ), and further in view of Browne et al . (US 2004/01 3581 5 A1 ) 
('Browne'). 

With respect to claim 27, Denoue teaches: 

receiving a path drawn on a display via user input (paragraphs 54 and 86), the 
drawn path defining boundaries of a selected on-screen region of the display, the 
selected on-screen region comprising a plurality of pixels (Figure 6, paragraphs 59 and 
67); 



Application/Control Number: 10/766,319 Page 19 

Art Unit: 2164 

capturing the plurality of pixels of the on-screen region (paragraphs 67 and 86- 
87); and 

storing the captured image pixels (paragraph 89); 

extracting a character or word from textual data as context information via the 
computer system (paragraph 79); and 

storing the extracted context information in association with the captured data 
(paragraphs 75-76). 

Denoue does not teach: storing captured data in an image file; automatically 
extracting a character or word from the textual data as context information in response 
to automatically determining that the displayed content of the on-screen region includes 
the textual data via the computer system; or storing the extracted context information 
and additional context information in association with the image file via the data storage 
subsystem, such that the context information is accessible when viewing the image file. 

Dawe teaches a system and method for saving handwriting as an annotation in a 
scanned document (see abstract), in which he teaches: 

capturing image pixels and storing captured image pixels in an image file (col. 3 
lines 60-63, col. 7 line 61- col. 8 line 4); 

automatically extracting a character or word from the textual data as context 
information in response to automatically determining that the displayed content of the 
on-screen region includes the textual data via the computer system (col. 7 lines 6-8 
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and 15-20); and 

storing the extracted context information and additional context information in 
association with the image file via the data storage subsystem, such that the context 
information is accessible when viewing the image file (col. 7 lines 57-60, col. 8 lines 1- 
16). 

It would have been obvious to a person having ordinary skill in the art at the time 
the invention was made to have modified Denoue by the teaching of Dawe to enable 
handwriting to be saved as an annotation, and thus reduce clutter in a document 
presented to a user, while maintaining the information conveyed by the handwriting for 
presentation to the user if desired (Dawe, abstract). 

Further regarding claim 27, Denoue in view of Dawe does not teach a displayed 
content of the selected on-screen region includes both textual data and underlying data 
comprising at least one of: an executable object, a file, or a link to remote content; or 
automatically extracting a property of the underlying data as additional context 
information in response to determining that the displayed content of the on-screen 
region includes underlying data via the computer system, the property comprising at 
least one of: a file name, a file identifier, a uniform resource locator (URL), a uniform 
resource identifier (URI), a folder name, or meta-data. 

Hertzfeld teaches a method and apparatus for context sensitive text recognition 
(see abstract), in which he teaches: 

a displayed content of the selected on-screen region includes both textual data 
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and underlying data comprising at least one of: an executable object, a file, or a link to 
remote content (Hertzfeld, steps 602 and 608 in Figure 6, paragraph 38); and 

automatically extracting a property of the underlying data as additional context 
information in response to determining that the displayed content of the on-screen 
region includes underlying data via the computer system, the property comprising at 
least one of: a file name, a file identifier, a uniform resource locator (URL), a uniform 
resource identifier (URI), a folder name, or meta-data (step 616 in Figure 6, paragraph 
38). 

It would have been obvious to a person having ordinary skill in the art at the time 
the invention was made to have further modified Denoue by the teaching of Hertzfeld 
because the content of the selected on-screen region includes both textual data and 
underlying data comprising at least one of: an executable object, a file, or a link to 
remote content; and in response to determining that the displayed content of the on- 
screen region includes underlying data, automatically extracting a property of the 
underlying data as additional context information, the property comprising at least one 
of: a file name, a file identifier, a uniform resource locator (URL), a uniform resource 
identifier (URI), a folder name, and meta-data would enable recognition of predefined 
types of text and predefined actions to be performed based on the types of text 
(Hertzfeld, abstract). 

Further regarding claim 27, Denoue in view of Dawe and Hertzfeld does not 
teach extracting complete sentences based upon punctuation. 



Application/Control Number: 10/766,319 Page 22 

Art Unit: 2164 

Chan teaches a system and method for providing bilingual annotations (see 
abstract), in which he teaches extracting complete sentences based upon punctuation 
(paragraph 77). 

It would have been obvious to a person having ordinary skill in the art at the time 
the invention was made to have further modified Denoue by the teaching of Chan to 
enable an annotation tool providing annotations in multiple languages, thereby making 
documents accessible to more users (Chan, abstract, paragraph 14). 

Further regarding claim 27, Denoue in view of Dawe, Hertzfeld and Chan does 
not teach pointing a first pointer from the context information to the displayed content; 
or pointing a second pointer from the additional context information to the displayed 
content. 

Browne teaches a method and apparatus for image metadata entry (see 
abstract), in which he teaches pointing a first pointer from the context information to the 
displayed content; and pointing a second pointer from the additional context information 
to the displayed content (paragraphs 136 and 142). {Browne teaches storing metadata- 
image (i.e. context-content) associations using pointers. He further teaches that 
metadata records comprise references (i.e. pointers) to images.) 

It would have been obvious to a person having ordinary skill in the art at the time 
the invention was made to have further modified Denoue by the teaching of Browne to 
enable an easy and efficient method of classifying and storing digital images, including 
images of text (Browne, paragraph 14). 
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With respect to claim 24, Denoue in view of Dawe, Hertzfeld, Chan and Browne 
teaches: 

digitizing movements of a stylus across the display in order to receive the 
annotation (Denoue, paragraphs 55, 57 and 75); and 

obtaining additional context information based on the received annotation 
(Denoue, paragraph 79), the additional context information being automatically stored in 
association with the image file (Dawe, col. 7 lines 57-65). 

17. Claims 28 and 31 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Dawe et. al (US 7,042,594 B1 , filing date 3/7/2000) ('Dawe') in view of Woolf et al. 
(WO 01/42980 A1 ) ('Woolf), further in view of Hertzfeld et al. (US 2002/00761 09 A1 , 
filing date 1/25/1999) ('Hertzfeld'), and further in view of Barsness et al. (US 
2004/0201633 A1 ) ('Barsness'). 

With respect to claim 28, Dawe teaches: 

receiving a display of content comprising at least one of textual data, an 
executable object, a file, or a link to remote content (col. 6 lines 42-48, col. 4 lines 46- 
59, col. 7 lines 38-40); 

receiving a path drawn on the display which defines boundaries of a selected on- 
screen region of the content via user input (col. 5 lines 28-30; col. 6 lines 42-51); 

capturing and storing the selected content as an image file via the data storage 
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subsystem (col. 3 lines 60-63, col. 7 lines 40-42 and lines 61-65, col. 8 lines 1-3); 

performing text recognition on a received annotation as context information (col. 
7 lines 6-20); 

automatically determining via the computer system that the content displayed 
within the on-screen region includes textual data and underlying data (col. 6 lines 52- 
53, col. 7 lines 1 1-20 and 35-40); 

automatically extracting as additional context information: 

a bounded region comprising a character or word from the textual data 
determined to be included in the on-screen region (col. 7 lines 6-20 and 57-65, col. 8 
lines 19-26), and determining if the character or word lies completely within the 
bounded region (col. 7 lines 11-16) (Dawe teaches that it is determined whether the 
region includes any handwriting using handwriting detection logic. Therefore, the 
handwriting detection logic determines if any handwriting (i.e. at least one character) 
lies completely within the region.); and 

storing the context information and the additional context information in 
association with the image file via the data storage subsystem, such that the context 
information is accessible when viewing the image file (col. 7 lines 57-60, col. 8 lines 1- 
16). 

Further regarding claim 28, Dawe does not teach setting a mode of operation to 
a content capture mode; switching the mode of operation to an annotation mode; or 
receiving an annotation on the display via the input device; or. 
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Woolf teaches a system and method for annotating an electronic document (see 
abstract), in which he teaches: 

setting a mode of operation to a content capture mode (page 7 lines 10-12); 

switching the mode of operation to the annotation mode; and 

receiving an annotation drawn on the display via the input device (Figs. 4 and 5, 
page 7 lines 5-12 and lines 24-26 and). 

It would have been obvious to a person having ordinary skill in the art at the time 
the invention was made to have modified Dawe by the teaching of Woolf because 
receiving an annotation drawn on the display would enable users the ability to annotate 
pages of a document without changing the contents of the document (Woolf, page 2 
lines 14-15). 

Further regarding claim 28, Dawe in view of Woolf does not teach extracting as 
additional context information, a property of the underlying data determined to be 
included in the on-screen region, the property comprising at least one of: a file name, a 
file identifier, a uniform resource locator (URL), a uniform resource identifier (URI), a 
folder name, or meta-data. 

Hertzfeld teaches a method and apparatus for context sensitive text recognition 
(see abstract), in which he teaches automatically extracting as additional context 
information, a property of the underlying data determined to be included in the on- 
screen region, the property comprising at least one of: a file name, a file identifier, a 
uniform resource locator (URL), a uniform resource identifier (URI), a folder name, or 
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meta-data (step 616 in Figure 6, paragraph 38). 

It would have been obvious to a person having ordinary skill in the art at the time 
the invention was made to have further modified Dawe by the teaching of Hertzfeld to 
enable recognition of predefined types of text and predefined actions to be performed 
based on the types of text (Hertzfeld, abstract). 

Further regarding claim 28, Dawe in view of Woolf and Hertzfeld fails to teach 
pointing a first pointer from the selected content to the context information; or pointing a 
second pointer from the selected content to the additional context information. 

Barsness teaches a handheld electronic book reader with annotation and usage 
tracking capabilities (see abstract), in which he teaches: pointing a first pointer from the 
selected content to the context information; and pointing a second pointer from the 
selected content to the additional context information (paragraphs 36, 49 and 61-62). 
(Barsness teaches a term is associated with a context list that has one or more context 
records 69, each context record including either a copy of a section of text or a pointer 
to the actual content in an electronic document. The term represents selected content, 
and each term may be associated, by pointers, to different contexts of the document.) 

It would have been obvious to a person having ordinary skill in the art at the time 
the invention was made to have further modified Dawe by the teaching of Barsness to 
enable improved functionality and utility of electronic documents by enhancing the 
collaborative and annotative capabilities of electronic book readers or other similar 
handheld devices capable of rendering electronic documents (Barsness, abstract, 
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With respect to claim 31 , Dawe in view of Woolf, Hertzfeld and Barsness teaches 
wherein the annotation is stored as originally drawn as additional context information in 
association with the image file (Dawe, col. 7 lines 9-1 1 , 18-20 and 57-65). 

Response to Arguments 

18. Applicant's arguments with respect to claims 22, 24, 27, 28, and 31 have been 
considered but are moot in view of the new ground(s) of rejection. 

1 9. Applicant's arguments filed September 3, 201 0, regarding claim 1 0, have been 
fully considered but they are not persuasive. Applicant argues that the Denoue and 
Dawe fail to describe two separate sets. The Examiner disagrees. Denoue teaches 
selecting content, such as "Increased Awareness" (paragraphs 67, Figure 6). He further 
teaches that the object "Awareness" may be selected alone as a standalone object, and 
that the content "Increased Awareness" may represent two parts (i.e. two sets of 
characters, "Increased" and "Awareness") (paragraphs 71-72). Therefore, Denoue 
teaches two separate sets of textual characters. 

Conclusion 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to ALICIA M. LEWIS whose telephone number is 
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(571)272-5599. The examiner can normally be reached on Monday - Friday, 9 - 6:30, 
alternate Friday off. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Charles Rones can be reached on 571-272-4085. The fax phone number 
for the organization where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the 
Patent Application Information Retrieval (PAIR) system. Status information for 
published applications may be obtained from either Private PAIR or Public PAIR. 
Status information for unpublished applications is available through Private PAIR only. 
For more information about the PAIR system, see http://pair-direct.uspto.gov. Should 
you have questions on access to the Private PAIR system, contact the Electronic 
Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a 
USPTO Customer Service Representative or access to the automated information 
system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 

/Alicia M Lewis/ 
Examiner, Art Unit 2164 
October 21, 2010 



